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The University of Nebraska at Kearney's Online Course Checklist is the main 
instrument for assessing the quality of online courses at UNK. A number of issues 
were faced when developing and deploying this quality assurance checklist at a 
small four-year university. The process including development, implementation, 
and revision is discussed along with the need for faculty buy-in. Special 
considerations included how to assess the quality of an online course while ensuring 
academic freedom in accordance with a union contract. The purpose of this case 
study is to provide a roadmap for institutions that are developing an instrument of 

their own. 

The number of students taking online courses continues to grow. 
According to a recent study, "Grade change: Tracking online education in the United 
States," by the Online Learning Consortium, 33.5% of higher education students in 
the fall of 2012 took an online course (Allen & Seaman, 2014). A total of 7.1 million 
students were taking at least one online course, an increase of 411,000 students 
from the previous year (Allen & Seaman, 2014). This represents a 6.1% increase 
(Allen & Seaman, 2014). Almost two-thirds of Chief Academic Officers (CAO) 
believe that it is very likely that students at higher education institutions will take at 
least one online course in the next five years. In addition, 65.9% of CAOs also 
believe online education is a critical piece in the long-term strategy for their 
institutions (Allen & Seaman, 2014). 

As the number of courses delivered online has continued to grow, 
knowledge of what constitutes a successful online course has increased as well 
(Khan & Chishti, 2012; Singer, 2008). These include, but are not limited to, 
navigation (Chen, 2010; Graf, Luit, & Kinshuk, 2010; Metros & Hedberg, 2002; 
Sung & Mayer, 2012), the inclusion of rich media components (Borgemenke, Holt, & 
Fish, 2013; Havice, Davis, Foxx, & Havice, 2010; Schiefelbein, 2011) and 
opportunities for collaboration with peers and the instructor (Brindley, Blaschke, & 
Walti, 2009; Gonzalez, 2010; Orde et al., 2001). 

Choices in course design have a far-reaching impact. The best medium 
through which to communicate within the confines of the learning management 
system must be selected (Schiefelbein, 2011). Planned flexibility in navigation 
helps learners adapt to the environment based on their cognitive styles (Chen, 
2009). Course design impacts student engagement and success (Murray, Perez, 
Geist, Hedrick, & Steinback, 2012), with scaffolding helping to increase student 
engagement (Cho & Cho, 2013) and learner satisfaction (Havice et al., 2010). 
Online videos are growing in importance as ease of delivery continues to improve 
(DeCesare, 2014). 
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The success of an online course is based— The success of an 

in part—on the quality of instructional design. online course is based— 

Faculty members are subject matter experts, but in part—on the quality 
may not have the pedagogical background and of instructional design, 
instructional design skills necessary to develop 

quality online courses (Caplan, 2004). By helping them understand and apply best 
practices in online course creation and delivery, there is an increased level of 
success for faculty building courses and the learners for whom they build the class 
(Murray et al., 2012). 

To address this, institutions have adopted various strategies to provide 
support for faculty members developing online courses. This may come in the form 
of course shells or templates (Miller, 2012; "With Seton Hall's Course Template, All 
Faculty Are Champions," 2011). Other institutions create programs that provide 
faculty instructional design support ("Instructional Designer's Guide to Working with 
Faculty," 2006; Lorenzetti, 2009) to help build competency. Models where faculty 
focus on course design within their own discipline, rather than focusing on higher- 
level instructional design (Power, 2008), have proven effective at some institutions. 

Some institutions focus on online course design from the institutional level, 
focusing on minimizing institutional barriers such as "compensation and time; 
organizational change; and technical expertise, support, and infrastructure" (Orr, 
Williams, & Pennington, 2009, p. 258). Others approach course development by 
creating an institutional infrastructure where online courses are an essential 
component of the institution's mission while shifting programs to online delivery 
(Hillman & Corkery, 2010). Online courses take more time to develop than 
traditional face-to-face courses (Rahm-Barnett & Donaldson, 2008). The amount of 
time and effort spent in developing these courses can cause pieces to be missed. 

Regardless of the process or approach, quality assurance can be used to 
build confidence in the process and ensure that key factors leading to the success of 
the online course are incorporated into the course structure. Developing standard 
process and procedures aids in identifying quality standards ("Holding on to quality 
during rapid expansion", 2011; Hosie, Schibeci, & Backhaus, 2005). The quality 
assurance instrument provides a blueprint to follow during the review process, 
helping to streamline the review process. 

Prebuilt quality control evaluation instruments such as Maryland Online's 
Quality Matters (QM) rubric, the Illinois Online Network's (ION) Quality Online 
Course Initiative rubric (QOCI), and California State University, Chico's Rubric for 
Online Instruction (ROI) may be adopted, while other institutions may consider 
building an evaluation instrument based on specific institutional needs. Such was 
the case at the University of Nebraska at Kearney where an in-house quality 
assurance instrument was developed. The purpose of this case study is to review 
the development, implementation, and revision process for UNK's Online Course 
Checklist and to provide a roadmap for any institution that is developing an 
instrument of their own. 

Background 

The University of Nebraska at Kearney (UNK) is a rural four-year university 
in the Great Plains. The 2013-14 total enrollment at UNK was 7,100 students with 
approximately 77% of these being undergraduates. eCampus is a UNK support 
department for online, blended, and videoconference courses. The department 
currently consists of nine full-time positions, including a director, three instructional 
designers, two video engineers, a marketing specialist, and two office associates. 
These nine positions support the approximately 100 faculty who teach online each 
semester. 

Prior to 2004, no training was offered to faculty who taught online. By 

2005, two instructional designers had been hired, and a training course was being 
offered once a year to faculty who wished to improve their online teaching. Prior to 

2006, online courses were not evaluated at UNK. Faculty created and taught online 
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courses with no evaluation process. As online learning continued to grow, so did 
the need for quality assurance. With the rapid growth of online courses at UNK, the 
next logical step was to create a way to ensure that best practices from the training 
course were being used in online courses. "When managing multiple simultaneous 
course development projects, online development units can identify quality 
standards by taking advantage of standard processes and procedures" ("Holding on 
to quality during rapid expansion", 2011, p. 5). An evaluation instrument would 
show where "design and delivery are sound" and "identify deficient aspects" within 
courses (Hosie et al., 2005, p. 545). 

Faculty at UNK develop their own courses, with the instructional designers 
available for consultation and to create media and interactive pieces that would be 
too difficult for the faculty to create on their own. This process has proven to be the 
most efficient way to support such a large number of courses with only three 
instructional designers. 

The instructional design team referenced in this article consisted of two 
people. Steven McGahan and Karen Premer were, at the time when the instrument 
was developed, the only two instructional designers on the UNK campus. It should 
be noted that the process described in this case study was informal, until the 
instrument was turned over for review by the committees. The instructional design 
team was still relatively new, and the development of the instrument was a bottom- 
up process. 

UNK had been offering online courses since 2000, when there were two 
courses with a total enrollment of 35 students. The first instructional designer was 
hired in December of 2003, and then a second was hired in October of 2004 to meet 
the growing demand for support for online courses. In the summer of 2006, online 
courses were the fastest growing segment of UNK course and program offerings. In 
the summer of 2006, there were a total of 52 faculty, offering 78 courses 
(undergraduate and graduate) with a total duplicated enrollment of 1,558. This 
substantial growth in online courses at UNK was one factor in the need for a way to 
assess the quality of courses. 

With two instructional designers to consult and assist faculty in their course 
development and a solid training program in place, the next logical step was to 
either find or develop an instrument to ensure the quality of the courses being 
developed at UNK. Instructional designers attended the Illinois Online Network's 
Faculty Summer Institute (FSI) to find new ideas and technology for use at UNK. 
The idea to create a quality assurance instrument began at FSI. 

One of the sessions was on the Quality Online Course Initiative Rubric, 
developed by the Illinois Online Network. This instrument rated the quality of online 
courses using many different criteria and allowed faculty to see where their courses 
might need improvement. Upon returning to UNK, the instructional designers 
decided to look into using one of the outside assessment instruments. The process 
began with the initial research into the rubrics available from other universities and 
initiatives. 


Initial Research 

Once the decision was made to apply a course quality instrument to UNK 
online courses, the first step was to review various instruments available to look for 
a prebuilt quality control evaluation that could be implemented immediately. With 
several options to choose from, the instructional designers narrowed the choices to 
three that were the most viable options and best fit the needs of the faculty: 
Maryland Online's Quality Matters (QM) rubric, the Illinois Online Network's (ION) 
Quality Online Course Initiative rubric (QOCI), and California State University, 
Chico's Rubric for Online Instruction (ROI). 
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Quality Matters 


The Quality Matters (QM) rubric, developed by Maryland Online, is one of 
the most well-known course evaluation rubrics in online education. This was the 
natural starting-point to begin investigating off-the-shelf evaluation rubrics for use 
at UNK. The QM rubric covers several areas of criteria, including course overview 
and introduction, learning objectives, assessment and measurement, instructional 
materials, learner interaction, and engagement, course technology, learner support, 
and accessibility ("Quality Matters rubric," n.d.). 

The official, fee-based process was reviewed first. The official process uses 
external reviewers to review a course. A QM reviewer assigns points based on the 
separate criteria that are weighted on a three-point scale. The weighted points 
denote the importance of each criterion, and the criteria that have a value of three 
points are all required to pass the review process. 

Quality Matters had some distinct advantages. It was a large and 
comprehensive course evaluation rubric. The QM rubric covered several critical 
areas that would later be used as a starting point for some of the UNK course 
evaluation rubric. It also showed levels of competency for each criterion and 
weighted them according to their importance to an online course. 

The disadvantages of the QM rubric were evident early in the review 
process. The complexity of the rubric and grading system precluded being used to 
assess online courses at UNK. The rubric also required a significant time 
commitment to complete, and there were not enough staff members to properly 
execute this grading rubric. Cost factors also contributed to not being able to use 
the official Quality Matters course evaluation process, as QM uses a fee-based 
system for their official reviews ("Guidelines from the Quality Matters rubric", 
2009). 

The use of the QM rubric would have been problematic because of concerns 
regarding academic freedom and compliance with the union contract governing UNK 
faculty. The academic freedom policy does not allow the grading of courses on a 
scale without some intervention by the Faculty Senate to approve this process. 
While this factor was unknown during the initial review (this concern would be 
brought up during the faculty review process), a way to evaluate the courses 
without assigning any points or a scale to the process was needed. 

Quality Online Course Initiative 

The Quality Online Course Initiative Rubric (QOCI) is an instrument 
developed by the Illinois Online Network. Much like the Quality Matters rubric, 
QOCI is a comprehensive rubric that covers several areas in the development of 
online courses, including instructional design; communication, interaction, and 
collaboration; student evaluation and assessment; learner support and resources; 
web design; and course evaluation ("Quality Online Course Initiative rubric", n.d.). 
The instrument uses a four-point grading system to evaluate criteria in each of the 
areas for quality. Unlike the QM rubric, QOCI is free to use, download, and modify 
for use at any institution (Patterson Lorenzetti, 2007), as it is licensed under a 
Creative Commons copyright ("Quality Online Course Initiative rubric", n.d.). 

Much like the other instruments, this comprehensive rubric did a good job 
at evaluating the quality of a course. Its comprehensive nature was a positive, just 
like QM. It covered many areas that are critical to a quality online course. Unlike 
QM, no complex grading system was in the rubric, and therefore, was easier to use. 

Again, there were disadvantages to QOCI. Much like Quality Matters, the 
instrument was rather long and took significant time to complete. QOCI comes in at 
23 pages without instructions. The amount of time necessary to complete the 
rubric for a course would have been difficult to manage with the number of new 
courses being developed at UNK. Had QOCI been adopted, faculty and instructional 
designers would have been spending most of their time going through the rubric for 
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each of the courses, and QOCI would not have been able to offer other training 
opportunities and multimedia development to the faculty. 

Again, the issues of evaluating faculty work and academic freedom would 
prohibit the adoption of this instrument. Information on how these issues were 
addressed will be presented at a later point in this discussion. 

Rubric for Online Instruction 

The Rubric for Online Instruction (ROI) is an instrument developed at 
California State University, Chico. The immediate opinion of the ROI was that at 
seven pages, it was far more compact than the other two instruments. Although it 
still covered many of the same issues in development, it would not take as much 
time to review a course as the other rubrics that were examined. The ROI covered 
six areas of development, including learner support and resources; online 
organization and design; instructional design and delivery; assessment and 
evaluation of student learning; innovative teaching with technology; and faculty use 
of student feedback ("Rubric for Online Instruction", n.d.). It was the easiest to 
use, and the time necessary to review a course was the shortest. ROI is also free to 
use and modify as it is available under a Creative Commons license ("Rubric for 
Online Instruction", n.d.). 

Of the three instruments that were reviewed, the Rubric for Online 
Instruction was the least comprehensive. While the time factor was important, it 
was still necessary to take a comprehensive look at the aspects of a course that 
were seen as the most important. The ROI was a little too simple for use at UNK. 
While the categories were fairly comprehensive, specifics within each were too 
broad and did not point out certain key areas that were important in UNK courses. 
The number of criteria in each of the six sections ranged from three to five, for a 
total of 25 criteria. 

One other drawback was the use of a three-point scale to evaluate each 
criterion. The ROI still used a rubric system that grouped each criterion into one of 
three classifications. This would certainly cause issues with the evaluation of faculty 
teaching and development. Again, it was necessary to use an instrument that did 
not appear to rate or rank the courses or faculty. 

Instrument Development 

After the determination was made that none of the three existing 
instruments met the specific needs of the UNK faculty and campus, it was decided 
that a specific UNK instrument was needed. Several factors needed to be addressed 
in an instrument for use on the UNK campus. These included ease of use, 
addressing multiple skill levels, encouraging rich media, and compliance with the 
faculty union contract. 

Development of the Instrument Criteria 

Culling from the large number of criteria that make a quality online course 
is not an easy task. The instructional designers reviewed several off-the-shelf 
instruments and current research on online course quality to determine what criteria 
would be included in the final instrument. Research shows that there are several 
factors to consider when developing a successful online course. Some of the major 
categories that were included were ease of course navigation, collaboration and 
interaction, encouraging rich media, multiple modes of learning and assessment, 
and Americans with Disabilities Act considerations. The instructional designers 
decided to leave out the more esoteric elements such as teaching style from the 
instrument, since these elements would be difficult to quantify. 
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Ease of Course Navigation 


Course navigation can easily turn great content into a bad learning 
experience. Students need to feel comfortable in moving around a course, and a 
confusing navigation system will cause issues with learning. It seems a simple idea; 
however, poor navigational structure can be easily overlooked. Emphasis needs to 
be placed on how learners interact with the course interface. There is a need for 
clear design in navigation and interface and for less complexity in online courses 
(Metros & Hedberg, 2002). Students need a well-structured and organized course 
to be successful in online courses (Hoffman, 2012). 

As with writing, a course developer can become so close to their own 
content that they are not able to see the issues present with the finished product. 
The course interface must be intuitive and facilitate learning and interaction 
between the learners and the content, instructor, and other students (Metros & 
Hedberg, 2002). What seems simple to use to the faculty member can feel like an 
impassible maze to the students. Course shells can improve navigation, leading to 
improved engagement (Miller, 2012). 

Considering the learning styles of students may also reduce effort as well 
(Graf et al., 2010). Based on the way learners view the course structure, they can 
get lost in the learning management system (Sung & Mayer, 2012). The demands 
of navigation can cause a cognitive overload, preventing them from learning the 
content. The inability to find content, assignments, or links to help resources may 
frustrate students in a course (Sung & Mayer, 2012). Navigation behaviors differ 
based on cognitive style (Chen, 2010). 

Several criteria were chosen to reflect the importance of navigation 
("eCampus online," 2014); these include: 

• A concise list of units/modules that will be completed in the course is 
provided; 

• A concise list of activities that will be completed in the course is 
provided; 

• The navigational instructions make the organization of the course easy 
to understand; 

• The content is arranged in a manner that enables learners to achieve 
the stated goals; 

• Resources and materials are easily accessible to and usable by the 
learners; 

• Consistent layout design is used throughout; 

• Scrolling is minimized by "chunking" materials; 

• Navigation cues are present and identifiable (i.e., Start Here). 

Collaboration and Interaction 

Collaboration and interaction with the instructor, the content, and other 
students is important in helping the students feel connected to the course (Poole, 
2000). Engaged students tend to be more successful students (Poole, 2000). 

Requiring interaction among students and between students and the 
instructor using coordinated group activities, such as discussion boards and 
synchronous class sessions, can be very effective for learning (Orde et al., 2001). 
The formation of small collaborative groups can lead to deeper learning (Brindley et 
al., 2009). Use of these types of interaction and collaboration, if done correctly, can 
facilitate higher-level learning, and students can broaden their knowledge by 
reflecting on their own experiences and the knowledge and experiences of the other 
students and the instructor (Gonzalez, 2010). These types of tools can also be used 
solely to present information and course content (Gonzalez, 2010). 

Collaboration and interaction were important aspects that needed to be 
addressed in the instrument. Not only do these create a sense of community in a 
course, but they also require that students learn along a specific timeline. In order 
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to participate in the course, students must interact with each other and the 
instructor at specific times during the course. 

Several criteria were chosen to reflect the importance of collaboration and 
interaction ("eCampus online," 2014); these include: 

• The instructor's role in discussion activities is clearly defined. 
(Facilitate, Clarify, Question, Observe, etc.). 

• Learning activities are developed to stimulate communication and/or 
collaboration between student and instructor. 

• Discussions are organized in specific forums and/or threads. 

• The requirements for course interaction are clearly articulated. 

• A statement is provided explaining when students should receive 
feedback for assignments, exams, discussion boards, and emails. 

• Learning activities are developed to stimulate communication and/or 
collaboration between student and student. 

• The course offers separate forums for Technical Questions, Content 
Related Questions, and Community Communication. 

• A rubric defining student participation is provided. 

Encouraging Rich Media 

Rich media is an important part of online courses. Rich media can provide 
learners with nuanced messages through verbal and non-verbal cues, as well as 
"spoken and written word, and visual symbols and images" (Schiefelbein, 2011, p. 
1). Simple text-based courses may have been standard in the early days of online 
education; however, students expect more from their online courses than reading 
and taking tests or writing papers. The integration of images, charts and graphs, 
audio, video, and interactive learning objects creates more interaction between the 
student and the content. In some cases, the inclusion of media itself isn't as 
important as the sense of control the learner feels in the environment (Borgemenke 
et al., 2013). The ability to control playback of rich media may help in "promoting 
greater reinforcement of course content and student engagement in the course" 
(Havice et al., 2010, p. 57; Lee & Chan, 2007). 

When the instrument was originally written, online video was still in its 
infancy, and most courses at LINK did not use much in the way of visual elements. 
It is sometimes hard to believe how quickly technology changes the way instruction 
and content are handled. Images and audio were the common rich media at that 
point in time. Because of this, the technical aspects were the primary focus of the 
instrument. 

The focus was not to make the courses pretty, but to add media that 
differentiated important areas and helped to engage the students in the course. As 
video was one of the more advanced aspects, it was necessary to steer these 
additions in the correct direction. The main purpose of the video should be to 
deliver content and methods of instruction, not to make the course more visually 
appealing (Castle & McGuire, 2010). This is not to say that these elements can only 
be used for presenting content. Increasing the presence and establishing the 
personality of the instructor can be achieved as well. Both synchronous and 
asynchronous video communication has an impact on students' view of the presence 
of the instructor and helps them perceive the instructor as a real person (Borup, 
West, & Graham, 2012). 

Several criteria were chosen to reflect the importance of rich media 
("eCampus online," 2014); these include: 

• Technical Requirements - no specialized software or hardware is 
needed, and all players are provided or available freely for download; 

• Images added to the course are clear; 

• Image files are optimized for efficient loading; 
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• Audio materials added to the course are clear; 

• Audio file length is adequate to meet the goals of the activity without 
adding unnecessary information; 

• Video materials added to the course are clear; 

• Video file length is adequate to meet the goals of the activity without 
adding unnecessary information; 

• The selection and use of tools and media supports the learning 
objectives of the course; 

• Selection and use of tools and media enhances learner interactivity. 

Development Factors for the Instrument 

In addition to focusing on the criteria for the instrument, it was necessary 
to consider specific issues that would arise when faculty at UNK tried to use the 
instrument. Some of these issues are common to most secondary educators; 
others were specific to the UNK faculty. When developing an online course 
evaluation, it is necessary to take into account not only what will be evaluated, but 
also how the instrument will function in the hands of those who use it. This meant 
looking at three specific issues: ease of use of the instrument, use by faculty with 
multiple skill levels, and compliance with the University of Nebraska at Kearney 
Education Association (UNKEA) Faculty Contract. 

Ease of Instrument Use 

First and foremost, the developed instrument needed to be easy for faculty 
to use. The main issue discovered when reviewing the off-the-shelf instruments 
was that the instruments were complex and required time-intensive evaluation. The 
instructional designers determined that they needed a rubric that would be both 
comprehensive and easy to use; the final instrument could be no more than three 
pages long and be easy enough to use as a self-assessment. 

In order to get faculty on board with using the instrument, it needed to be 
simple enough to use without the assistance of an instructional designer. An 
instrument like Quality Matters or the Quality Online Course Initiative rubric 
requires a great deal of time and effort to complete successfully. These instruments 
included some instructional design principles that may have been unfamiliar to 
some faculty at UNK. Building and teaching an online course is time-consuming 
enough without adding a long process to decide whether the course is up to a 

certain standard of quality. The QM rubric has 
When developing an online 41 separate quality criteria on one sheet; and, 

course evaluation, it^ is while this may seem like a quick system to use 

necessary to take into for the review of a course, it also prescribes 

account not only what will weighted measures for each criterion with a 

be evaluated, but also how secondary system that requires the reviewer to 

the instrument will function calculate a total score based on these weighted 

in the hands of those who measures. QOCI ascribes a more standard, non- 

use weighted system to assessing course quality; 

however, it also requires that the reviewer go 
through 23 pages of criteria and rate each on a three-point scale. 

Thus, the UNK instrument needed to be a hybrid of these two ideas: short 
enough to be completed in a relatively small period of time, and comprehensive 
enough to ensure that certain quality standards were addressed. Eventually, a 
simple complete/not complete checklist format was used, instead of a rubric-style 
rating system. This system made it much faster to complete than trying to assign a 
level of skill to each criterion. It also allowed the faculty to see what was needed 
for a quality course without having to read in-depth rationale for why a particular 
criterion was important, and read through examples of what was considered to be 
unsatisfactory versus excellent, and ratings in between. Although not as 
comprehensive, this system also allowed the faculty to take ownership of their own 
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work, as it relied on their experience as instructors to determine which factors 
constituted the level of quality. This relies on the faculty (with additional help from 
an instructional designer, when needed) to bring their experience with teaching to 
the instrument. 

Addressing Multiple Faculty Skill Levels 

Online courses are constantly in a state of change. As new content, 
technology, and resources arrive, courses must be changed to reflect the current 
state of the materials and of the resources available to the faculty who develop 
these courses. eCampus continually offers new technology and resources to assist 
faculty in the development and teaching of online courses. Faculty at UNK also 
have different levels of expertise with online education. Therefore, the instructional 
design team felt that one inclusive rubric would not satisfy the needs of all faculty. 

To ameliorate the situation, instructional designers decided to break the 
instrument up into three separate instruments reflecting a higher skill level for each 
successive version. The first instrument would cover the basics of building an online 
course for the first time. This covered basics like course overview materials, basic 
navigation, technical support, course management, grading, and assessment. 

The second rubric was built to function for the first revision of the course. 
When faculty were preparing to teach the course again, they would use the second 
rubric to add more features and raise the quality of the course. Criteria represented 
in the second rubric included broader use of multimedia, chunking materials, 
inclusion of rubrics, more depth in interaction and discussion, and more access to 
resources. 

The final rubric covered more advanced areas, such as full ADA 
compliance, multiple learning methods, and more advanced materials to foster 
student-content interaction. 

Compliance with Union Contract 

One of the issues the instructional designers encountered when developing 
the online course evaluation instrument was the faculty union contract at UNK. The 
contract does not allow the evaluation of faculty because of academic freedom. 
Thus, faculty are allowed to develop their courses in whatever manner they wish. 
Balancing course quality with academic freedom was the key idea in creating a 
checklist instead of a rubric. 

Since faculty at UNK cannot be directly evaluated, the checklist format, at 
the very least, allowed faculty to see some of the basic criteria that go into a quality 
course. This method is far less intrusive than an instrument that would tell them 
they had failed to create a quality course. Instead of showing faculty that they do 
not have a quality course, it shows where they may have missed items that can be 
included to create a quality course. While this system may not be perfect, it was 
the best that could be created to comply with the union contract and also ensure a 
level of quality. The use of the word "rubric" versus "checklist" will be discussed 
further in the following section. 


Faculty Buy-In 

When the first draft of the instrument was finished, it was necessary to 
gain the support of the key stakeholders in online course development. This meant 
that eCampus needed support from faculty on the use of the instrument. The 
logical place to start was the UNK Faculty Senate. Rather than take the instrument 
to the full senate, it was presented to the eCampus committee to gain support, 
resolve issues, and gain feedback. 

The initial presentation of the UNK Online Course Rubric to the eCampus 
Faculty Senate committee was in late 2006. The committee members were given 
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copies of the rubric and asked to give their feedback on the rubric as a whole and 
on each of the individual criteria. The reviews were mostly positive, but some 
issues needed to be addressed. 

The most strenuous arguments against the instrument in the form of a 
rubric were due to factors associated with academic freedom and the union 
contract. Committee members expressed concern over using a rubric to evaluate 
faculty: "While I think the rubric is comprehensive, I'm concerned that using a 
rubric violates our academic freedom." The general consensus was that the Union 
Contract did not allow for faculty to be assessed in this manner. 

Several members of the committee were adamantly against using an 
evaluation rubric. One faculty member vehemently stated, "I don't use rubrics in 
my classes, why should I have to use one to evaluate my online course?" This 
statement reflected the attitudes of many faculty on the UNK campus at that time. 
Online education was still relatively new, and there was push-back about assessing 
teaching. This has changed over the past several years, but the legalities have 
remained the same. It should also be noted that there were, and still are, some 
faculty who resent having instructional designers with master's degrees evaluating 
the work of those with doctorates. 

After further debate, the committee decided to change the rubric to a self- 
assessment instrument in the form of a checklist. While not ideal, this solution also 
created more buy-in from the faculty. At this point, the rubric officially became a 
checklist. 

The members of the eCampus Faculty Senate Committee gave further 
input on minor changes in wording. These changes were considered, and then most 
were implemented, as none of them caused the criteria to change significantly. 
Some examples include the use of "learning activities" instead of "assessment," the 
inclusion of "modules" instead of "units" to refer to learning units, and adding 
"provided software" to a statement about freely available downloads. Afterward, 
the instrument received approval to move forward to the full Faculty Senate for a 
vote on implementation. The Online Course Checklist was approved by the Faculty 
Senate for use at UNK in the spring of 2007. 

Testing the Instrument 

Once the checklist was approved for use, a testing phase was initiated to 
determine if the instrument was useable, both by the instructional designers as a 
review instrument and by the faculty as a self-assessment. The time-intensive 
nature of the testing process for the checklist did not allow for a quantitative 
assessment of the instrument. Instead, the testing process was qualitative. The 
initial testing phase was an informal process of using the checklist during online 
course reviews and consultations. This worked as a self-selecting process where the 
first faculty to come for a consultation or course review were asked to use the 
checklist as part of the process. A total of 17 faculty members participated in the 
testing phase and provided feedback. Additional faculty teaching online were asked 
to review the checklist on their own to assess its usability as a self-assessment tool; 
these included members of the eCampus Faculty Senate Committee. 

Initial reaction by the faculty was positive. The checklist helped to identify 
areas in their courses that were deficient and allowed them to revise their 
navigation and content to reflect a more user friendly experience for both the 
students and the faculty. The most common areas in the anecdotal evidence from 
part one of the checklist were those of navigation issues and the availability of 
resources. The second checklist showed the two criteria most commonly missed 
were "chunking" of materials and consistent use of navigational cues. The most 
common issue found when using the third checklist was the lack of proper ADA 
compliance. 

Feedback on the checklist was not totally positive. A few faculty members 
voiced concerns about academic freedom, which was a common thread running 
through the creation process of the checklist. Some faculty were not happy about 
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having a set of standards applied to their teaching. The use of the checklist to 
ensure quality was seen as an affront to their ability as instructors, even though the 
transition to an online format created new issues that they had not encountered in 
their face-to-face courses. The addition of a technical aspect, while seen as a 
threat, did not register as an area that some faculty felt needed any oversight. 

One of the more common issues was that of length. Some felt that the 
instrument was too long to be used as a self-evaluation. This reinforced the idea to 
split the checklist into three separate parts, instead of presenting it as a whole 
instrument. After a review of the criteria, it was decided that, while there would be 
some complaints from faculty, the comprehensive nature of the instrument should 
stay the same, and none of the criteria would be eliminated. This also reinforced 
the instructional designers' choice not to use the off-the-shelf instruments reviewed 
initially. 


Revision of the Instrument and Deployment 

Once the testing process was completed, the checklist was revised to 
reflect some of the concerns. The checklists did not change greatly during the 
revision process; mainly, language was revised and items were moved around in 
the individual checklists and between checklists. Some criteria were rewritten to be 
more specific, and others were moved to the more advanced checklists and vice- 
versa. 

Once the revision was complete, it was necessary to deploy the checklists 
for use by the faculty and the instructional designers. The checklists were initially 
deployed on the eCampus website providing easy access to those who wanted to 
use it. An announcement regarding the availability of the checklists was sent to the 
entire campus via email. In addition to these electronic distribution methods, 
several information and training sessions were held to demonstrate and discuss the 
checklists with faculty. Finally, any faculty member who applied for a stipend to 
create an online course was sent a copy of the initial checklist via campus mail. 

It is worth noting that the three checklists have recently been reduced to 
two. After several years of use, it was decided that the general level of expertise of 
the UNK faculty had grown enough to move some of the more advanced criteria to 
the first and second checklists. This left only a few criteria on the third checklist, so 
these were moved to the second checklist as well. This revision of the original 
instruments was also the impetus for writing this case study. 

Integration with Other Resources 

One of the initial outcomes of the testing and revision phase of the project 
was the need to integrate the checklist with other resources. Specifically, the 
Online Course Template was developed to give faculty a starting point for their 
classes and add in resources that many would not include in their course design. 

The course template adds a Unit/Module structure to the traditional 
Blackboard shell and allows faculty to break up their content over several areas, 
instead of putting it all together in one folder. This creates an easier to understand 
and less confusing navigation structure for students to follow. 

These particular features were addressed in several of the checklist 
sections. The use of these review standards in the checklist tied it to the template. 
Many of the criteria from the first checklist were tied directly to the template. This 
reinforced the standards and also made it easier for faculty to comply with the 
checklist criteria. This also allowed the instructional designers to perform a course 
review in less time, provided the faculty member had used the template in their 
course. Tying together the checklist and template also promoted the resources 
available to faculty across the disciplines. 
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Key Takeaways 


Building a course evaluation instrument can take large amounts of time 
and resources. Some best practices to help reduce this time are as follows: 

• Use a pre-built instrument if it fits the needs of the campus. 

• Review several instruments and apply them to a few courses before 
deciding what to use. 

• If creating an instrument, start with instruments that fall under 
Creative Commons, rather than starting from scratch 

• Identify the three or four most important issues that must be 
addressed and start with these. 

• Once the most important issues are addressed and complete, move to 
the next set. 

• Make sure that the instrument is easy to use and has language that 
does not confuse the reader. Involve faculty (and union if there is one 
on campus) to create a higher level of faculty buy-in. 

• Test, test, test. There is no substitute for testing the instrument on 
multiple courses with multiple reviewers. Some of these reviewers 
should be faculty. 

• Align the instrument with training and other resources. 

Conclusion 

An online course evaluation instrument is a critical component to any 
successful online course/program/institution. While prebuilt instruments can be 
quickly deployed, they may not properly serve the local campus community. 
Careful review of these instruments is needed before making any decision about 
creating a specific instrument for a campus. This may take time and resources, but 
it will create a more successful faculty, which in turn, creates a more successful 
student population. Tailoring an instrument to a specific university or college can 
be an excellent use of resources, if done with care and thought. 

Recommendations 

While reaction to the Online Course Checklist has been mostly positive, 
some issues should be addressed in the future. These include, but are not limited 
to, creating a separate actionable checklist for proactive development, 
recoding/grouping the criteria to make the checklist more useable, and revising the 
checklist to include newer technology and pedagogy skills. 

The Online Course Checklist has 
been a useful evaluative tool for online 
courses at UNK. As an actionable tool for 
course development, there are some 
doubts as to its efficacy. The specific 
nature of the criteria makes the checklist 
better for reactive changes to courses. A 
smaller instrument, with broader categories and fewer granular items, might be 
more useful as a tool for initial creation of courses. This type of instrument would 
create a roadmap for those creating a course for the first time. 

Another issue is that the checklist has not been updated recently. 
Developments in technology and learning research may necessitate the revision of 
the instrument to reflect current standards and best practices. Most notably, new 
collaborative technology to facilitate interaction, the expanded use of interactive 
content to engage students, and the greater role of both in-house and professional 
video as a means of instruction and engagement. This should be an ongoing 
process, as a checklist of this nature should be a living document that reflects the 
current state of online education. 


Make sure that the instrument is 
easy to use and has language that 
does not confuse the reader. 
Involve faculty (and union if 
there is one on campus) to create 
a higher level of faculty buy-in. 
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